Skip to content

fix/mcp-taxonomy-pagination - #1

Merged
Formatted merged 1 commit into
mainfrom
Formatted-patch-1
Apr 8, 2026
Merged

fix/mcp-taxonomy-pagination#1
Formatted merged 1 commit into
mainfrom
Formatted-patch-1

Conversation

@Formatted

Copy link
Copy Markdown
Owner

## What does this PR do?

Fixes a silent data-loss bug (MemPalace#171) in the four MCP taxonomy tools (mempalace_status, mempalace_list_wings, mempalace_list_rooms, mempalace_get_taxonomy). On palaces with more than ~10k drawers, all four functions called col.get(limit=10000) in a single shot and wrapped the whole thing in except Exception: pass. Two things could go wrong:

  1. ChromaDB silently caps or returns None for metadatas on large collections, causing a TypeError: 'NoneType' is not iterable that gets swallowed — result: {"wings": {}} on a 96k-drawer palace.
  2. Even when it doesn't raise, a single 10k-item fetch misses everything beyond the cap.

The fix extracts a shared _iter_metadatas(col, where=None) generator that pages through the collection in batches of 500 (the same pattern already used correctly in layers.py), guards against None metadatas at each page, and logs a warning instead of silently swallowing errors. All four taxonomy functions are updated to use it.

## How to test

# Install deps
pip install -e ".[dev]"

# Run the new pagination tests (no network needed — uses in-memory fixtures)
python -m pytest tests/test_mcp_server.py::TestTaxonomyPagination -v

# Full suite
python -m pytest tests/ -v

The three new tests in TestTaxonomyPagination monkey-patch _TAXONOMY_BATCH = 2 to force multi-page fetching on a small fixture, then assert all items are counted across pages. A third test confirms graceful handling when ChromaDB returns None for metadatas.

## Checklist

  • Tests pass (python -m pytest tests/ -v)
  • No hardcoded paths
  • Linter passes (ruff check .)

@Formatted
Formatted merged commit 02dcbb9 into main Apr 8, 2026
@Formatted
Formatted deleted the Formatted-patch-1 branch April 8, 2026 03:21
Formatted pushed a commit that referenced this pull request May 8, 2026
- Resolve UU conflict in hooks_cli.py: take develop/HEAD approach
  (mine synchronously via _mine_sync, then pass through unconditionally).
  _mine_sync already catches subprocess.TimeoutExpired — fixes Copilot #1.
- Add tests/test_palace_locks.py: 4 tests covering mine_global_lock
  non-blocking semantics (acquire, second-acquire raises MineAlreadyRunning,
  reusable after release, release on exception) — fixes Copilot MemPalace#4.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Formatted pushed a commit that referenced this pull request May 8, 2026
…ut, tests

Addresses the six Copilot review comments on the initial commit.

1) MemPalace#6 (critical) — mcp_server.py `_get_collection` bypassed ChromaBackend

   The MCP server creates its palace collection directly via
   `chromadb.PersistentClient.get_or_create_collection` in `_get_collection`,
   not through `ChromaBackend.get_collection`. That path was missing the
   `hnsw:num_threads=1` metadata, so the primary crash surface for MemPalace#974
   and MemPalace#965 was untouched by the original patch. Fixed by passing
   `hnsw:num_threads=1` at the mcp_server create site too. Documented
   in a code comment that the setting is only honored at creation
   time — existing palaces created before this fix still need a
   `mempalace nuke` + re-mine to gain the protection.

2) MemPalace#3 — mine_global_lock over-serialized mines across unrelated palaces

   Replaced the single global lock file `mine_global.lock` with a
   per-palace lock keyed by `sha256(os.path.abspath(palace_path))`
   (`mine_palace_<hash>.lock`). Mines against the same palace still
   collapse to a single runner (the correctness boundary), but mines
   against *different* palaces are now free to run in parallel.
   `mine_global_lock` is kept as a backward-compatible alias for
   `mine_palace_lock` so any external callers that imported the
   previous name keep working.

3) #1 — hook_precompact swallowed OSError but not subprocess.TimeoutExpired

   `subprocess.run(..., timeout=60)` raises `TimeoutExpired` on slow
   palaces. The previous `except OSError` clause didn't catch it, so
   the hook could raise and fail to emit any JSON decision — leaving
   the harness without a block/passthrough signal. Fixed by catching
   `(OSError, subprocess.TimeoutExpired)` together and always falling
   through to the block decision so the hook reliably emits a response.

4) MemPalace#2 + MemPalace#4 — tests

   - tests/test_hooks_cli.py: added
     `test_precompact_first_two_attempts_block`,
     `test_precompact_passes_through_after_cap`, and
     `test_precompact_counter_is_per_session` to lock in the MemPalace#955
     deadlock fix.
   - tests/test_palace_locks.py (new): covers `mine_palace_lock`
     single-acquire, reuse-after-release, cross-process serialization
     on the same palace, non-interference across different palaces,
     path normalization, and the `mine_global_lock` back-compat alias.

5) MemPalace#5 — known limitation, documented but not auto-fixed

   Copilot suggested detecting collections missing `hnsw:num_threads=1`
   and calling `collection.modify(metadata=...)` to retrofit existing
   palaces. Verified against chromadb 1.5.7: `modify(metadata=...)`
   replaces metadata rather than merging, and re-passing
   `hnsw:space="cosine"` then raises `ValueError: Changing the
   distance function of a collection once it is created is not
   supported currently.` The HNSW runtime configuration
   (`configuration_json`) also does not expose `num_threads` in
   chromadb 1.5.x, so the flag appears to be read only at creation
   time. Rather than paper over the limitation with a best-effort
   `modify` that silently drops `hnsw:space`, documented in the
   mcp_server comment that pre-existing palaces need a
   `mempalace nuke` + re-mine to gain the protection. Fresh palaces
   are always protected.

Testing
- pytest tests/test_palace_locks.py tests/test_hooks_cli.py
  tests/test_backends.py tests/test_cli.py → **98 passed, 0 failed**.
- Runtime validation with two concurrent `mempalace mine` calls:
  - Different palaces → both complete in parallel ✓
  - Same palace     → one completes, the other exits with
    "another `mine` is already running against <palace> — exiting
    cleanly." ✓
Formatted pushed a commit that referenced this pull request May 8, 2026
The MCP `mempalace_get_drawer` tool returned the entire raw drawer
metadata blob to any connected client, and the `source_file` field
in that blob is the absolute filesystem path written by the miners
(`miner.py`, `convo_miner.py` — `source_file = str(filepath)`). On
a single-user local deployment this is self-disclosure, but in
nested-agent or multi-server MCP topologies the client is a separate
trust domain and the host's directory layout has no documented
client-side use.

Mirror the mitigation that `searcher.search_memories()` already applies
on its own return path: reduce `source_file` to its basename via
`Path(source_file).name` before handing the metadata to the client.
Citations still work — the directory layout does not leak.

Companion to #1 (omit palace_path from tool_status). Same threat class,
different surface:

- mempalace_status — palace dir path     → fixed in #1
- mempalace_get_drawer — per-drawer source_file path → this PR

Other read tools were audited and do not leak host paths:
- mempalace_search    — already basenames source_file
- mempalace_list_drawers — returns wing/room/preview only
- mempalace_diary_read   — date/timestamp/topic/content only
- mempalace_reconnect    — success/message/drawers only
- mempalace_kg_*         — entity/predicate strings, counts
- mempalace_check_duplicate — wing/room/preview only

Changes:
- mempalace/mcp_server.py: tool_get_drawer() now basenames metadata.source_file
- tests/test_mcp_server.py: regression test asserting the absolute path
  and its parent directory do not appear anywhere in the response
- website/reference/mcp-tools.md: clarify the documented return shape
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant